Sample tito_session_mismatch instead of running it on every record fetch#63
Open
nightlessbaron wants to merge 1 commit into
Open
Sample tito_session_mismatch instead of running it on every record fetch#63nightlessbaron wants to merge 1 commit into
nightlessbaron wants to merge 1 commit into
Conversation
…fetch Re-tokenizing the full trajectory on the session-server event loop blocks record retrieval at long context; gate it behind --tito-session-mismatch-sample-rate (default off). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
rmfan
approved these changes
Jul 1, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
On agentic runs,
get_sessionre-tokenizes the full trajectory to computetito_session_mismatchon every record fetch, synchronously on the session-server event loop. At 64k-context SWE scale this blocks record retrieval — the client-sidecollect_recordsGET times out (120s), returns empty, and the sample gets marked ABORTED. On a recent run that was ~19k timeouts, ~58% of samples per step, which cascaded into whole groups being dropped by the infra-failure filter.The mismatch is a diagnostic canary for chat-template / tokenizer drift, logged as
tito_session_mismatch_rate. It is not a training-correctness guard — training tokens come straight from the inference response (prompt_token_ids/output_token_logprobs), and the per-turn alignment asserts on the training path already enforce that. So computing it on every session is pure overhead once the template is stable.This gates it behind
--tito-session-mismatch-sample-rate(default0.0, off). Set0.02for a cheap drift signal,1.0to restore the previous always-on behaviour. Sampling is a deterministiccrc32(session_id)so there is no RNG on the hot path.